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Smdv Approach 

This study by the American Productivity & Quality Center (APQC) was commissioned 
by Loral Space Information Systems, Inc. and the National Aeronautics and Space 
Administration (NASA) to evaluate internal assessment systems. APQC benchmarked 
approaches to the internal assessment of quality management systems in three phases. 

The first phase included work conducted for the International Benchmarking Clearinghouse 
(IBC) and consisted of an in-depth analysis of the 1991 Malcolm Baldrige National Quality 
Award criteria. 

The second phase was also performed for the IBC and compared the 1991 award criteria among 
the following quality awards: Deming Prize, Malcolm Baldrige National Quality Award, The 
President’s Award for Quality and Productivity Improvement, The NASA Excellence Award 
(The George M. Lowe Trophy) for Quality and Productivity Improvement and the Shigeo 
Shingo Award for Excellence in Manufacturing. 

i 

The third phase compared the internal implementation approaches of 23 companies selected 
from American industry for their recognized, formal assessment systems. 

Major Study Findings 

1. All of the five quality awards considered in this paper have major differences in their award 
criteria and scoring guidelines. While the 1991 Baldrige Award Criteria were considered the 
most complete of the five awards, there is still room for improvement in the areas of process 
control and quality results. 

2. The weighting of the categories among the various awards indicates their focus for applicant 
recognition: The Shingo Prize recognizes companies that focus on process control at the shop 
floor level with half of its points awarded in this area, while the Baldrige Award, President's 
Award and NASA Excellence Award, recognizes a broader area of quality performance. The 
Deming Application Prize uses a prescriptive criteria system but applies the same criteria to all 
areas of the company to achieve breadth of application for its criteria. 

3. The 1992 Baldrige Award Criteria have strengthened the Examination Items noted as areas 
for improvement in paragraph 1 above. 
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4. Of the 23 companies whose internal assessment programs were evaluated, five used 
self-assessment surveys, one used a combination of self-assessment survey with an internal 
Baldrige-like assessment, nine used the Baldrige Criteria and approach without modification, 
and eight customized the Baldrige process or criteria for their own internal application. 

5. The biggest area for improvement in internal company assessment systems is the calibration 
of individual examiner scoring. One company reported point spreads greater than 400 points 
among examiners. This is consistent with the scores of individuals who have been initially 
selected for the Malcolm Baldrige National Quality Award (MBNQA) Board of Examiners, but 
who have not yet attended training. One intent of the MBNQA examiner training is to calibrate 
the Board of Examiners on recognizing the 50% level of performance using the scoring 
guidelines. 
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Study Objective 


Background 

In 1987 President Ronald Reagan signed the Malcolm Baldrige National Quality 
Improvement Act which established a national award to recognize quality improvement among 
manufacturing, service, and small businesses. The act did not describe the scoring system, 
judging process or criteria for evaluation of applications. These criteria have become an 
operational definition of Total Quality Management (TQM) and the wide distribution of the 
application guidelines has exposed many senior managers to the "Baldrige" definition of TQM. 

The Malcolm Baldrige National Quality Award was not the first prestigious quality 
award. That distinction goes to the Deming Application Prize of the Union of Japanese 
Scientists and Engineers. Initiated in 1951 and named after the W. Edwards Deming, the 
American quality guru who helped to begin the Japanese quality movement, the Deming Prize 
has been long recognized as an indicator of excellence in business. In a 1983 study by Dr. 
Noriaki Kano of Deming Prize recipients business performance compared with that of 
non-Deming Prize companies, it was observed that the Deming Prize winners had a 3-6% range 
of advantage in annual return on net assets, a measure of business profitability, over the 
non-winners from the same industry during the decade of the 1970's. 1 

While the Deming Prize is focused on statistical process control as the fundamental 
building block of quality (see the summary of the award criteria in Appendix A), the Baldrige 
Award applies customer satisfaction as the foundation of quality and applies quality methods to 
business management Other quality and productivity awards have been introduced to recognize 
improvement in particular areas, and have slightly different intent and criteria. These other 
awards include: 

♦ The President's Award for Quality and Productivity Improvement 

♦ The National Aeronautics and Space Administration Excellence Award for Quality and 
Productivity (The George M. Lowe Trophy) 

♦ The Shigeo Shingo Prize for Excellence in Manufacturing 

These awards have stimulated interest in a United States business community which 
thrives on competitive recognition. David A. Garvin, a Professor of Business Administration at 
the Harvard Business School and a member of the Board of Overseers for the Malcolm Baldrige 
National Quality Award from 1988 to 1990, has described the Baldrige Award as "the most 
important catalyst for transforming American business." 2 The award provides a framework for 
management to assess their progress for achieving quality results that produce competitive 
performance. Indeed, the General Accounting Office (GAO) has published a report that 
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evaluates the business competitiveness of 20 companies who were high scorers on the 1988 or 
1989 Baldrige Award applications. While the results of the GAO study were limited by the 
sample size, the GAO did conclude that there is a cause and effect relationship between the 
TQM practices embodied in the Baldrige Award Criteria and business performance as measured 
in terms of market share, productivity, and customer satisfaction. 3 

Purpose 

This study was conducted by the APQC's International Benchmarking Clearinghouse 
staff in conjunction with the APQC Consulting Group. It was commissioned as a project to help 
the National Aeronautics and Space Administration assess their approach and criteria for their 
Excellence Award for Quality and Productivity (The George M. Lowe Trophy.) The study 
analyzes the inter-relationships among the evaluation items of the Baldrige Award Criteria; 
assesses the similarities and differences among the various quality award criteria; and compares 
the approaches of different companies for incorporating internal self-assessment methods into 
their quality improvement approach. 


Benchmarking Methodology 

This study was conducted by evaluating open literature and presentations made by companies at 
quality forums where they presented the details of their quality assessment approach. This study 
uses information that was presented from 1989 through the present and may not reflect the most 
current approach of any particular company. However, the company information is considered 
to be representative of self-assessment methods using the Baldrige Award criteria approach. The 
study findings include both a matrix comparing company approaches and a summary of best 
practices which contains a detailed description of a model approach for integrating the Baldrige 
Award criteria into a company-wide assessment program. 


Organizations Evaluated 

The International Benchmarking Clearinghouse is a sendee operated by APQC to 
improve business competitiveness through a network of organizations dedicated to sharing 
improvement opportunities through benchmarking. During the design phase of the 
Clearinghouse, a survey of the 87 companies involved in establishing this benchmarking 
network was conducted. It was found that 74% of these companies were using the Malcolm 
Baldrige National Quality Award criteria for self-assessment, even though only 51% planned on 
applying for the award within the next five years. Of these companies 88% also ranked their 
competitive position as leaders in most of their markets. 4 Companies using benchmarking and 
applying the Baldrige Award criteria clearly perceive themselves as leaders in their respective 
markets. 
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The Baldrige Award Process 


The Intention of the Baldrige Award 

The Malcolm Baldrige National Quality Award is an annual Award to recognize U.S. 
companies that excel in quality management and quality achievement. The Award promotes: 
awareness of quality as an increasingly important element in competitiveness; understanding of 
the requirements for quality excellence; and, sharing of information on successful quality 
strategies and the benefits derived from implementation of these strategies. Up to two Awards 
may be given each year in each of three eligibility categories: manufacturing companies, service 
companies, and small businesses. The Award examination evaluates applicants according to a 
set of Criteria and scoring guidelines which are included in the Application Guidelines. These 
Criteria are designed to be a quality excellence standard for organizations seeking the highest 
levels of overall quality performance and competitiveness. These Criteria are reviewed and 
improved on an annual basis to reflect lessons learned during the evaluation process. The award 
is managed by the National Institute of Standards and Technology (NIST) and administered by 
the American Society for Quality Control (ASQC). The evaluation of applicants is conducted by 
a Board of Examiners who are nominated from the quality experts of business, professional and 
trade organizations, accrediting bodies, universities, and government. Members of the Board of 
Examiners must meet the highest standards of qualification and peer recognition. The Board of 
Examiners evaluates each application considering the context of the applicant's business factors, 
according to the Award Criteria, following a prescribed evaluation process, and using an 
established scoring guideline. Each of these elements of the Baldrige Award process is 
described in the following sections. 

The Effect of Business Factors 

While the Baldrige Award Criteria have been designed for their general application for 
evaluation of any company's quality system, independent of size, type of business, or market 
environment, it is recognized that the importance of individual business factor for a given 
company may influence the applicability of the items and areas to address, even for businesses 
of the comparable size or in the same industry. To give appropriate consideration to these 
distinctions, the application requests a four-page Overview, which does not count toward the 
page limit, that addresses key business factors that must be considered during the Award 
evaluation process. These business factors set the context for the interpretation of the entire 
application and are exceptionally important Information that is important to consider as 
business factors include: 

♦ Business structure of the applicant and relationship to parent company (if a subsidiary) 

Note: Subsidiaries should also include information that shows key relationships 
to the parent company: (1) percent of employees; (2) percent of sales; and 
(3) types of products and services. 

♦ Size and resources of the applicant 

♦ Types of major products and services 
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♦ Key quality requirements for products and services 

♦ Nature of markets (local, regional, national, or international) 

♦ Description of principal customers (consumers, other businesses, government) 

♦ Competitive environment 

♦ Applicant's position in the industry 

♦ Major equipment and facilities used 

♦ General description of the applicant's employment base, including: number, type, and 

education level 

♦ Importance of and types of suppliers of goods and services 

♦ Occupational health and safety, environmental, and other regulatory considerations 

♦ Other factors important to the applicant 

The Evaluation Process 

The Baldrige Award evaluation process contains four stages: first stage review of the 
written application; consensus review of the written application; site visit review; and Judges 
final review. Each applicant receives a feedback report which describes assessment of the Board 
of Examiners as to the Strengths and Areas for Improvement for each Examination Item in the 
application. The feedback report states observations and evaluations, not prescriptions on how 
to improve the applicant's quality process. The feedback report is prepared after an application 
has been eliminated from further consideration at one of the four stages of review. 

The first- stage review of the written application is conducted as an independent review 
by 5 or more members of the Board of Examiners. Each Examination Item is graded in 
accordance with the Scoring System. The Board of Examiners take the application at face value. 
They accept the facts as presented and, when questions arise, they record them for verification or 
clarification at the site visit. The Scorebooks from each evaluation are returned to NIST/ASQC 
and the Judges select the top-scoring applications for the consensus review. The written 
comments of the Examiners from the first stage review are used as the basis of the feedback 
report for applicants not selected for the consensus review. 

The consensus review of the higher- scoring applications from the first stage review is 
conducted by a team of 5 Examiners lead by a Senior Examiner. Consensus is initiated by the 
team leader with a goal of achieving an agreed upon scoring value for each of the Examination 
Items after the team has debated the relative merits of the applicant's approach, deployment, and 
results for that item. This is a particularly important step in the Award process because 
consensus numerical scores play a major role in determining which applicants will receive a site 
visit and in determining issues for review during the site visit In addition, the written comments 
of the Examiners from the consensus review are used as the basis of the feedback report for 
applicants that are not selected for site visits. Using these written comments, the Senior 
Examiner prepares a consensus report to the Judges, and the Judges select the top-scoring 
applicants for site visits. 

Finalists in the Award process receive a site visit from a team of six members of the 
Board of Examiners, lead by a Senior Examiner. The team visits one or more sites (labs, plants, 
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offices) of the applicant to clarify uncertain points in the application and to verify that the 
information presented by the applicant is correct. There are five distinct steps in the site visit 
process: 

1. Notification - The Panel of Judges selects applicants for site visits and transmits the 
information to the Award Administrator who notifies the applicant and the appropriate 
members of the Board of Examiners. 

2. Initial preparation - The team leader works with the NIST observer, Award 
Administrator, team members, and the point of contact from the applicant to establish the 
agenda and logistics for the site visit. Members of the site visit team perform review 
evaluation materials and perform tasks as assigned by the team leader. 

3. Final preparation - The team holds a day-long preparatory meeting immediately 
preceding the site visit. They finalize the agenda and category assignments, review site 
visit issues, and prepare the site visit worksheet which addresses the issues that the team 
will evaluate. This meeting is not held at the applicant s site. 

4. Conduct of the site visit - The actual site visit begins with an initial meeting with the 
applicant. During this meeting the team leader presents the agenda and objectives. After 
this introductory meeting, the site visit team divides and performs the individual 
Category assignments. The team caucuses as often as necessary to ensure that all 
assignments are being implemented, all issues/questions are being adequately addressed, 
and that the schedule is being followed. When the team leader and members are satisfied 
that all issues have been clarified or verified, the team leader closes the site visit by 
holding a meeting with the appropriate applicant representatives. 

5. Preparation of the site visit report — The site visit team completes the site visit worksheet 
and a report of their findings and conclusions about each Category, including the 
strengths and areas for improvement. The team leader reviews these worksheets for 
completeness and prepares a "Recommendation to the Judges" worksheet. 

Following the receipt of the site visit reports, the Judges meet for the final review to 
verify that the Baldrige Award process was followed, review the site visit reports, and make 
recommendations of the Award recipients. The Award Administrator forwards the Judges 
recommendations to the Secretary of Commerce for selection. 

Interpretation of the Criteria 

Some consulting companies have hinted that there is a hidden agenda in terms of what 
the Judges and Examiners are seeking in a "Baldrige-winning" company. But, there are no 
secrets. One member of the Board of Examiners has even published a book which presents his 
version of the interpretation of the Examination Items and what the Examiners evaluate. The 
Board of Examiners is seeking a company which represents a national role model for quality 
based on that company's approach and deployment of their quality program as well as the results 
that are attributable to that program. The essential elements of the Award Criteria are described 
in the Application Guidelines. Together, the following key concepts and core concepts define 
the infrastructure for the requirements of the Examination Items: 
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♦ Customer Driven Quality — Quality is defined by the customer. Business fundamentals, 
such as design quality and defect or error prevention, which affect the customer should be 
part of the quality system. 

♦ Leadership — The senior leadership of businesses must create clear quality values and 
build these values into the way the company operates. 

♦ Continuous Improvement -- Quality excellence derives from well-designed and 
well-executed systems and processes. Continuous improvement must be part of the 
management of all systems and processes. Companies need to communicate quality 
requirements to suppliers and distributors and work as a team to improve the performance 
of the entire business. 

♦ Fast Response — Shortening the response time of all operations and processes of the 
company needs to be part of the quality improvement effort. 

♦ Actions Based on Facts, Data, and Analysis -- Companies need to develop goals, as well as 

strategic and operational plans to achieve quality leadership. Operations and decisions of 
the company need to be based upon facts and data. 

♦ Participation by All Employees - All employees must be suitably trained, developed, and 
involved in quality activities. 

The 1991 Baldrige Award Criteria are summarized in Appendix B. It is important to 
understand the context in which these criteria are used. The Baldrige Examination is a two-part 
diagnostic system. The first part is the Criteria which includes the seven Categories which are 
divided into Areas to Address as well as the detailed description of these areas which are called 
the Examination Items. Taken together, the Criteria represent the "what" is to be evaluated. The 
second part of the diagnostic system, the Scoring System, represents the "how" evaluations are 
made. 


The Examination is non-prescriptive. While it is based on the values and key concepts 
described above, the Examination does not prescribe the specific means (specific techniques, 
methodologies, or organizational structure) to demonstrate excellence. The Examination also 
emphasizes the integration of the entire quality system. The Examination Items represent a 
system of requirements. Thus, quality system integration is the result of a company establishing 
the linkages among the direct and indirect relationships between the Examination Items. A 
coherent quality system demonstrates how these linkages are put into practice. In the Findings 
of this benchmarking study, the relationships among the Examination Items are demonstrated 
using an Affinity Diagram. 13 


Scoring Guidelines 

The Scoring System is based on three evaluation dimensions: approach, deployment, and 
results. All Examination Items require applicants to provide information or data relating to one 
or more of these dimensions. Each of the Examination Items is graded according to a scheme 
which considers the applicant's approach, deployment of that approach, and results demonstrated 
from that deployment. The specific interpretation of these dimensions is found in the following 
operational definitions. 
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"Approach" refers to the methods an organization uses to achieve the purpose described 
in the Examination Item. The Examiners consider such aspects of the approach as. 

♦ The degree to which the approach is prevention-based 

♦ The appropriateness of the tools, techniques, and methods chosen to meet the requirements 

♦ The effectiveness of the use of the tools, techniques, and methods 

♦ The degree to which the approach is systematic, integrated, and consistently applied 

♦ The degree to which the approach embodies effective evaluation or improvement cycles 

♦ The degree to which the approach is based upon quantitative information that is objective, 
timely and reliable 

♦ The utilization of unique and innovative approaches, including significant and effective 
new adap tations of tools and techniques used in other applications or types of business 

"Deployment" refers to the extent to which the approaches are applied to all relevant 
areas and activities which are either addressed or implied in the Examination Items. The 
Examiners consider such aspects of the deployment as: 

♦ The appropriate and effective application to all product and service characteristics 

♦ The appropriate and effective application to all transactions and relationships with 
customers, suppliers of goods and services and the public 

♦ The appropriate and effective application to all internal processes, activities, facilities, and 

employees 

"Results" refers to the outcomes, effects, and achievements that are attributable to the 
approach and deployment based upon the purposes addressed and implied in the Examination 
Items. The Examiners consider such aspects of the deployment as: 

♦ The quality levels are demonstrated and supported by evidence 

♦ The contributions of the outcomes and effects to quality improvement 

♦ The rate of quality improvement 

♦ The breadth of quality improvement 

♦ The demonstration of sustained improvement 

♦ The significance of improvements to the company's business 

♦ The comparison with industry and world leaders 

♦ The company's ability to show that improvements derive from their quality practices and 
actions 

Examiners are trained to recognize companies that score at the 50% level of performance 
using these scoring guidelines. Their training calibrates this level and, through the consensus 
grading process, reinforces the appreciation for excellence which is demonstrated by 
performance beyond this level. A 50% performance means that a company has a sound, 
systematic, prevention based approach which includes on-going improvement with evidence of 
integration; the approach has been deployed to most major areas of the company, and positive 
result trends that are caused by the approach are demonstrated in major areas. 
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Lessons Learned from the Baldrige Award 


Some of the lessons learned observed from the Baldrige Award are listed below using the 
Baldrige Categories to present them as a state of quality overview: 

1.0 Leadership 

♦ Senior management recognizes that quality is a strategic business issue. 

♦ Executives are communicating a quality vision and an accompanying value system to their 
employees and building them into their company culture. 

♦ The employment market for senior quality management positions indicates that the 
management structure of organizations are building an infrastructure to strategically 
deploy quality efforts. 

♦ Senior executives are speaking out for quality in public forums throughout America. 

♦ Senior management is not convinced ofthe relationship of quality performance driving 
financial performance - financial measures, rather than quality measures and the need for 
long-term management, still drive their perception of company performance. 

♦ Few senior managers use "management by fact" and apply the concept of process thinking 
to their own activities. 

♦ Cross-functional involvement in quality planning is beginning to be seen as a necessary 
activity. 
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♦ Planning for quality improvement is delegated to lower levels of the organization than the 
top management team. 

2.0 Information and Analysis 

♦ Information technology and data bases exist in most companies. 

♦ Benchmarking and sharing of information have increased greatly as a result of their 

emphasis in the Baldrige Criteria, however benchmarking is often confused with industrial 
tourism and few companies obtain the full value of this tool. 

♦ Resources are required to implement long-term measurement systems for product quality 
monitoring and customer and employee satisfaction surveys. The economic environment 
has precluded some of these investments. 

♦ Data collected are not distributed and communicated to all parties who require the 
processed information. 

3.0 Strategic Quality Planning 

♦ While benchmarking is becoming a more visible tool, few companies are integrating it 
with their strategic planning process. 

♦ The "Six Sigma" stretch goal of Motorola has received great publicity and many 
companies are considering how to improve their goal-setting capabilities. 

♦ Companies have poor communication methods to share their strategic goals and plans and, 
therefore, the organization is not in alignment with these goals. 

♦ Measurement of the effectiveness of the company planning system is not a common 
quality management practice. 

4.0 Human Resource Utilization 

♦ Team activity has greatly increased and suggestion systems have been implemented to gain 
more input from employees, although participation tends to be low. 

♦ While training budgets have increased substantially, the training tends to be basic and the 
effectiveness of training is not directly measured. 

♦ Employee surveys are used to assess employee morale, however, first level management 
resists the empowerment of their employees, perceiving it as a risk to job security. 

♦ While recognition has increased, recognition by presenting team quality improvement 
results to senior management is not greatly exercised. 

♦ Quality of work life and the ergonomic design of working conditions are not integrated 
into quality programs. 

5.0 Quality Assurance of Products and Services 

♦ Manufacturing is a strong area for quality — particularly for quality teams, statistical 
process control, just-in-time manufacturing, and supplier quality management. 

♦ Companies are beginning to seriously use design of experiments, Taguchi methods, and 
quality function deployment to enhance their product design and development processes. 
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♦ The spectre of the ISO 9000 registration requirements of the European Economic 
Community has increased management's interest in basic quality assurance. 

♦ While quality improvement is strong in manufacturing, corrective action tends to be 
symptomatic rather than focused on the root cause. 

♦ New product development is an area for improvement for most companies -- especially the 
customer listening systems needed to align the product design and delivery system with the 
voice of the customer. 

♦ Quality audits are not rigorously conducted in many manufacturing companies and are not 
conducted in service companies. 

♦ Quality efforts tend to focus on manufacturing and fail to involve service, support, and 
business process areas. 

6.0 Quality Results 

♦ Quality levels are improving in many industries. 

♦ Supplier quality programs, ISO 9000 and the Baldnge Award are all tending to improve 
the quality of the manufacturing supplier base. 

♦ Many companies do not know which processes are their key business processes and 
therefore do not focus on their improvement. 

♦ Product quality measurement is favored greatly over service quality measurement or 
customer-perceived quality performance. 

7.0 Customer Satisfaction 

♦ There is an increased focus on quick response to customers and formal complaint 
resolution, however management tends to be overdependent upon complaints as a source 
of customer feedback. 

♦ People in support functions are just beginning to understand the concept of the "internal" 
customer. 

♦ While customer contact people receive motivational training, they are not fully empowered 
to resolve issues and are reluctant to escalate the issue to the level of the individual with 
the authority to make the resolution. 

♦ The process for integrating customer data with new product design and development is 
informal in most companies. 

♦ Customer segmentation tends to be incomplete, not addressing all layers of customers 
(such as distribution channel or final consumer). 

♦ While the use of customer surveys is increasing, there is little technical understanding of 
the appropriate application of survey results. 

♦ Companies change their performance measurements frequently, producing a fragmented 
historical data base which is not able to make a comparison of trends over time. 

♦ Companies tend to believe that replacement guarantees should satisfy the customer, rather 
than providing preventive action. 

♦ Complaint systems are reactive and responding to formal complaints, rather than 
aggregating complaints from all sources and dealing with them as an integrated complaint 
management system. 
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Comparison of 1991 Quality and Productivity Award Criteria 

There are five major quality awards that have been presented to American companies: the 
Deming Application Prize, the Malcolm Baldrige National Quality Award, the President's 
Award for Quality and Productivity Improvement, the NASA Excellence Award (George M. 
Lowe Trophy), and the Shingo Prize for Excellence in Manufacturing. This section of the study 
describes the particulars of each award and compares the various award criteria and scoring 
systems. 


The Deming Application Prize 


The Deming Prize was established in 1951 to honor the contributions of Dr. W. Edwards 
Deming to the quality control movement within Japan. The Deming Application Pnze is 
awarded in three categories: Deming Application Prize for Division, Deming Application Prize 
for Small Business, and Quality Control Award for Factory. In addition, individuals who have 
uniquely contributed to Japan's body of knowledge about quality control and statistical methods 
may be awarded a Deming Prize. Any company that qualifies for the Deming Application Prize 
will receive it -- the prize is awarded without external competition and there is no maximum 
number of companies who may receive the award in a given year. 

To qualify for the Deming Application Prize, top management must apply. This is called 
challenging the Deming Prize. The process to receive the award lasts three to five years and the 
company’s management must convince the Deming Prize Committee that they are prepared for 
an on-site examination. These experts serve as examiners and audit the state of the quality 
system, paying particular attention to the use of statistical methods and using a brief set of 
"particulars" called the Deming Prize Application Checklist (Appendix A). To qualify for the 
award, a company must score 70 points or more, top management must score at least 70 points, 
and no unit of the company may score less than 50 points. Companies that have applied for the 
prize receive a report of the comments and recommendations of the Deming Pnze Committee 
which contain the findings about the desirable and undesirable aspects of their quality operations 
and constructive suggestions for change. 

The Malcolm Baldrige National Quality Award 

The Malcolm Baldrige National Quality Award was established by President Ronald 
Reagan in 1987 to honor Malcolm Baldrige, the former Secretary of the Department of 
Commerce. The Baldrige Award has three categories for application: manufacturing, service, 
and small business. The Baldrige Award is competitive among the annual applicants and only 
two awards may be given in each category annually, however, the Board of Examiners may elect 
not to present an award in a particular category during a given year. 

To qualify for the Baldrige Award, top management must apply. While the process to 
receive the award lasts one year from the time of application to the time of award announcement, 
it may take a company three to five years, or more, to develop a quality system that is 
competitive for the award. The application for the Award is limited to 75 single-sided pages for 


Al.1-13 


Measuring Quality Progress 


the two large business categories and 50 pages for small businesses. To qualify for the Award, 
the applicant goes through an extensive process (see the above section on the Baldrige process). 
To be a contender for the award a company should be capable of scoring well above 700 points 
on the application. The highest score to date on the application has been in the mid-800 point 
range. However, the Baldrige Award is not granted solely on the competitive score. A more 
subjective assessment by the Judges is also made to evaluate the potential for the applicant to 
serve as a national role model for quality improvement. Each company that applies for the 
Baldrige Award will receive a feedback report that describes the findings of the Board of 
Examiners relative to the company's strengths and areas for improvement. 

The President's Award for Quality and Productivity Improvement 

The President's Award for Quality and Productivity Improvement was established by 
President George Bush in 1988 to recognize quality and productivity improvements among 
agencies of the Federal Government. An agency becomes eligible to apply for the President's 
Award if one or more Quality Improvement Prototypes (QIPs) have been selected from that 
agency. The Quality Improvement Prototype Award is given to smaller units within an agency 
that have made significant improvements in quality and productivity. The criteria for these 
Awards are contained in the Federal TOM Handbook (The President's Award is summarized in 
Appendix C). The criteria for the QIP Award are a sub-set of the President's Award Criteria. 
While the President's Award may be given to two agencies annually, QEP Awards may be given 
up to six governmental units. 

To qualify for the Award top management must apply; however, unlike the Baldrige 
Award, applications are mailed to eligible agencies by the Federal Quality Institute. Like the 
Baldrige Award, the President's Award cycle is one year, however, this does not indicate the 
amount of time that it will take a government agency to become competitive for receiving the 
Award. The application for the Award is limited to 35 single-sided pages for agencies under 
20,000 employees and 60 pages for larger agencies. To qualify for the Award, the applicant is 
evaluated by a Panel of Judges using the scoring guidelines for the Award. Only two President's 
Awards have been presented to date, so information about the competitive range is not available. 
The guidelines for the award indicate that scores in the range of 80 - 100% are considered to be 
World Class. As a measure of comparison 40-60% scores in the Award Criteria Categories 
indicate an organization with a sound, well-implemented program. Like the Baldrige Award, the 
score is not the sole determinant of the consideration for the President's Award. A more 
subjective assessment by the Judges is also made to evaluate the potential for the applicant to 
serve as a role model to government agencies for TQM implementation and quality 
improvement. 


The NASA Excellence Award (George M. Lowe Trophy) 

The NASA Excellence Award precedes the Baldrige Award by 3 years. It was initially 
established in 1984 by James Beggs, the Administrator of NASA, to honor those companies who 
have contributed to the success of the nation's aerospace efforts and to encourage superior 
quality and productivity in the aerospace industry. The NASA Excellence Award was renamed 
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the George M. Lowe Trophy in 1991 in memory of a 27-year NASA veteran and an early 
pioneer in the development of the NASA Space Programs. The Award is presented in two 
categories: large and small business. The Award is not competitive and may be given to as 
many applicants as demonstrate the level of excellence required over the period of time 
specified. 

Top management must decide to apply for the Award and submits a letter of nomination 
with a brief statement of eligibility compliance. If selected by the Evaluation Committee to 
compete, the applicant then completes a 35 page application in response to the Award Criteria 
(summarized in Appendix D). The report guidelines requires that data covering a three year 
performance window be provided in the report. The Evaluation Committee reviews the 
applications and selects finalists to receive site visits. The Scoring Guidelines for the NASA 
Award are considerably different from the Baldrige and President s Awards. While the NASA 
Scoring Guidelines describe excellence as scores in the 91 - 100% range, the scores of Award 
recipients range between 800 and 900 points. All award applicants may request a debriefing to 
identify strengths and areas of improvement. Debriefings are conducted either face-to-face or by 
teleconference. 


The Shingo Prize for Excellence in Manufacturing 


The Shingo Prize for Excellence in Manufacturing was established in 1988 to honor 
Shigeo Shingo who, with Taiichi Ohno, was the co-creator of the revolutionary manufacturing 
techniques, methods, and processes which make up the Toyota Production System. This Prize 
recognizes companies and plants in the United States that have demonstrated outstanding 
achievements in manufacturing processes, quality, productivity enhancement, and customer 
satisfaction. The Shingo Prize is awarded in two categories for large and small businesses and 
only two awards may be given in each category. In addition, individuals who conduct 
professional research (whether in industry, academia or consulting) in the field of manufacturing 
excellence may submit a paper for consideration in a competition for a Shingo Research Prize. 
The Research Prize has three categories: professional, graduate students, and undergraduate 
students. Up to three awards may be given annually in each category. 


To qualify for the Shingo Prize, top management must apply. The Shingo Prize follows 
the same examination process as the Baldrige Award. The first stage of the award is the 
evaluation of the written application to the Prize Achievement Criteria (Appendix E). The 
Board of Examiners makes their award recommendations following their site visits to the 


finalists to the Shingo Prize Council. The decisions of the Prize Council are final. Companies 
that have applied for the Prize receive a report citing notable accomplishments and opportunities 
for possible improvement within their manufacturing systems. 


Comparison of Award Criteria 


These five awards have similarities in the way that they recognize improvement in 
company performance in the areas of quality and productivity. All of these awards recognize 
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the need for management attention (Baldrige Criteria 1,0), teamwork and empowerment 
(Baldrige Criteria 4.0), and quality assurance (Baldrige Criteria 5.0), which produce results. The 

NASA Award and the Shingo Prize do not specifically address information and analysis 
(Baldrige Category 2.0) or strategic quality planning (Baldrige Category 3.0). The Deming 
Prize Application Checklist does not specifically address customer satisfaction (Baldrige 



Category 7.0). One interesting observation from this study is that the Shingo Prize, with only 
slight changes could apply equally well to a service company. 15 

The above table references each of the various award criteria to the seven Baldrige 
Award Criteria and provides an assessment of the strength of the criteria for that category. Note 
that the most complete award, relative to the Baldrige, is the President's Award. This is not 
surprizing since the President's Award was developed following the Baldrige Award. The 
Shingo Prize achievement criteria are strong relative to the Baldrige Award, but they are not as 
complete. The NASA Excellence Award Criteria are not as complete nor as robust as the 
Baldrige Criteria, with the exception of the human resources area (section 2.2 in the NASA 
Criteria). The Deming Prize particulars are stronger than the Baldrige in Categories 5.0 and 6.0, 
but not as complete in all of the other areas. All of the Baldrige Categories are addressed by the 
Deming Prize particulars except for Category 7.0 (Customer Satisfaction). Section 5.0 was 
judged as weak in the Baldrige since it does not have a strong focus on cycle time and waste 
reduction. (Companies that pursue ISO 9000 certification or integrate ISO 9000 with the 
Baldrige Criteria will eliminate this perceived area for improvement.) Section six was judged as 
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Comparison of Category Weighting 



Derning 

Prize 

MBNQA 

President's 

Award 

NASA 

Award 

Shingo 

Prize 

Scoring Basis 

Prescriptive 

Non-Prescriptlve 

Non-Prescriptive 

Non-Prescriptlve 

Non-Preacriptive 

Criteria Definition 
Stability 

Fixed 

Continuously 

Improving 

Continuously 

Improving 

Continuously 

Improving 

Continuously 

Improving 

Point Maximum 

100 

1000 

200 

1000 

100% 

Cate$ 

WeiG 

jories 

hted 

NO 

YES 

YES 

YES 

YES 

Weighting 
Relative Category 

i 

200 

100 

100 

220 

100 

2 

100 

70 

75 
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0 

3 

100 

60 

75 


0 

4 

100 

150 

150 

180 

100 

5 

400 

140 

150 

250 

500 

6 

100 

180 

250 

120 

200 

7 

0 

300 

250 

230 

100 


weak in the Baldrige since it does not provide a strong assessment of business results. (Note that 
this area was changed significantly in the 1992 Baldnge Award Criteria.) 

Comparison of Category Weighting 

The quality system of a quality award should exhibit the characteristics of both customer 
satisfaction and continuous improvement. Customer satisfaction means that the applicants for 
the Award feel that they have been accurately and fairly judged and that the assessment provides 
them with value in terms of increased self-knowledge of their quality system's strengths and 
areas for improvement. Continuous improvement means that interpretation of the criteria 
changes and scores achieved from year to year are difficult to compare. Scoring an application 
is probably the most emotional aspect of the assessment process. The first aspect of scoring is 
the weighting of the categories. The Table above shows an assessment of the various Award 
systems and their approach to weighting their categories after using the categories normalized to 
the Baldrige Criteria from the previous section. 

Note that the Derning Prize is the only award that is prescriptive and has a fixed criteria 
definition. The Derning categories are not formally weighted, however, many areas of the 
company are graded and the entire grade sheet is used in each area. The purpose of each award 
is reflected in the way that the categories are weighted. Most notably, see the Shingo Prize 
where applicants can score a maximum of 500 points in a single category — manufacturing 
improvement, likewise the Derning Prize gives 400 points in process control while the Baldrige 
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gives 300 in customer satisfaction. The weighting of the categories is used to reflect the 
behavior desired in the successful applicants. 

Comparison of Award Scoring Systems 

Interestingly, neither the Deming Application Prize nor the Shingo Prize have a formal 
scoring system. The Baldrige Scoring system was described earlier. A matrix is included in the 
Application Guidelines to show how the Examiners are to assign scores to particular items 
depending on the approach/deployment/results for that particular Examination Item. The 
scoring systems for the President's Award is so specific in its definitions of behaviors, that it 
borders on a prescriptive approach. The NASA scoring guidelines are skewed to the high side 
since 71-80% of the points in a category can be obtained by "gradual continual improvement." 
The NASA Award is unique in that it uses a specified duration for the system being in place, 
percent of deployment, resource allocation levels for quality program, and the degree of 
planning integration as indicators of the scoring. The NASA Award is the most complicated of 
the scoring systems and appears to be an area for improvement 

Analysis of Alternative Approaches to Internal Assessment 

The 23 companies studied in this analysis could be grouped into four categories based on 
their approach to the administration of the assessment criteria. These categories include: 
self-assessment only, self-assessment plus MBNQA assessment, MBNQA assessment, and 
custom internal assessment program. These categories and their ratings are shown below: 

1. Self - Assessment: 5 

This class of company used a survey or set of forms to conduct their internal Baldrige 
assessment. 

2. Self - Assessment plus MBNQA Assessment: 1 

This company uses the self assessment to screen divisions before they apply for an 
internal award. 

3. MBNQA Criteria Used for Internal Assessment: 9 

These companies followed the Baldrige Examination Process and used the Baldrige 
Criteria. 

4. Companies developed customized approach: 8 

These companies adapted the MBNQA Criteria to their company's culture. These 
companies included: Control Data Corporation, Hewlett Packard, IBM Europe, Intel, 
Proctor & Gamble, Westinghouse, Whirlpool, and Xerox. 

Although two of the standard MBNQA-clone company awards offered multiple award 
levels (e.g., bronze, silver, and gold), there was not much originality in the MBNQA look-alike 
awards. The unique applications for internal assessment are found in the custom category. Each 
of the unique items in these assessment programs will be described under the "best practice" 
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area. First, a brief note on "best practice." In the case of internal assessment, it only goes back 
about 8 years, Westinghouse having the earliest internal assessment program. Other than a few 
of the "quality regulars" there has been little in-depth revision of the Baldrige Criteria. Since 
this is the case, there is too little data to call a practice best. So, they will be considered as 
interesting processes or, at least different approaches. In addition, the single company that 
combined self-assessment with MBNQA evaluation approach will be discussed. 

Best Practices and Unusual Observations 

♦ Company A: This company used a self-assessment survey by all of the divisions as a 
screening mechanism to determine finalists for their internal award program. Then they 
asked these finalists to prepare an application and go through an MBNQA process to 
compete for the company-wide award. 

♦ Company B: This company does not use the Baldrige Criteria at all, but uses two radar 
diagrams to display their management and product quality profiles. On the management 
profile they have eight dimensions: Planning, recognition, participation, trust, 
cooperation, environmental, people management, and acceptance of new ideas. On the 
product profile they have six dimensions: price, service, reliability, functionality, user 
friendliness, and documentation. These reviews are conducted by a cross-functional 
management team. 

♦ Company C: This company customized the Baldrige Categories to their own cultural 
norms to define world-wide excellence: substituting fact-based management for 
information and analysis; strategic planning for strategic quality planning; 
COMPANYNAME people for human resource utilization; quality of processes and 
products for quality assurance of products and services; and measurement & results for 
quality results. Both leadership and customer satisfaction labels stayed the same. The 
areas to address under each of these seven areas was then defined in terms of the 
company's programs and culture. Otherwise, this company followed the MBNQA 
process. 

♦ Company D: This company translated the seven Baldrige Categories into five categories 
which they call the five areas of quality management: planning process, customer focus, 
improvement cycle, process management, and total participation. Each of these different 
categories was uniquely defined for the company's quality program and culture. The 
MBNQA scoring system was used for the evaluation; and results were displayed using a 
radar diagram with these five categories comprising an operating unit's quality profile. 

♦ Company E: This company merged the MBNQA Criteria with ISO 9000 and their own 
long-standing "reliability essentials" program to create a composite set of evaluation 
criteria. 

♦ Company F: This company uses a twelve category evaluation criteria, called the 
conditions for excellence, for their internal assessment process: customer orientation, 
participation, development, motivation, products and services, processes and procedures, 
information, suppliers, culture, planning, communications, and accountability. The 
company has also changed the weighting system for the categories and uses a unique 
scoring system. Another twist this company puts on die Baldrige process is the use of 
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♦ external examiners and judges for their assessors. Otherwise, their method follows the 
Baldrige process. 

♦ Company G: This company uses the Baldrige process to determine eligibility for their 
internal company award. After a division has scored 500 points on the written first stage 
application, then they are eligible for a site visit where they are evaluated on the six 
dimensions of the company's values: results orientation, risk taking, discipline, customer 
satisfaction, quality, and a great place to work. The administrative processes follow the 
Baldrige Award. 

♦ Company H: This company has not decided to Stan an award process, but has challenged 
each operating unit to use the Baldrige Criteria to achieve a "Baldrige certification" 
which occurs when they score over 750 points as verified by a one day site visit from a 
team of examiners. This challenge is not too burdensome from the paperwork viewpoint 
since the site visit is principally oral. This company has basically used a Deming scoring 
system with the Baldrige Criteria. 

♦ Company I: This company imposed a twenty page limit on applicants and applies a 
unique award criteria that is tied to their operational definition of TQM. Eligibility for 
the award includes business units, divisions, teams, and individuals. Three questions are 
evaluated in the application: why was the problem chosen; how was the problem 
approached; and what results were obtained and how significant were there. Each of 
these three questions is scored according to the contribution of: innovation, leadership; 
product, and service. 

Biggest Area for Improvement 

The biggest problem area outstanding is the scoring habits of the examiners. In one 
company’s training of internal examiners they found point spreads in scoring to be greater than 
four hundred points, which implies that the examiners needed more training to calibrate their 
observation skills. 
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"I would lay it down as a basic principle of human organization that the 
individuals who hold the reins of power in any enterprise cannot trust 
themselves to be adequately self-critical. For those in power the danger 
of self-deception is very great, the danger of failing to see the problems or 
refusing to see them is ever-present. And the only protection is to create 
an atmosphere in which anyone can speak up." 

John W. Gardner 

How to Prevent Organizational Dry Rot 
Harper's Magazine 
October 1965 


Produced by the staff of the International Benchmarking Clearinghouse in 
conjunction with the APQC Consulting Group. This paper was edited by 
Gregory H. Watson, Vice President of Benchmarking Services at APQC. Mr. 
Watson served as a member of the 1991 Board of Examiners for the Malcolm 
Baldrige National Quality Award. 


NOTICE 

This study contains information developed under internal 
research and development and therefore protected by copyright. 

The tables describing the comparative analyses of quality prizes 
and the affinity diagram of the Baldrige Criteria may not be photocopied 
or reproduced in any form without prior written consent from the: 
American Productivity & Quality Center 
123 North Post Oak Lane 
Houston, TX 77024 
(713) 681-4020 
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